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DETAILED ACTION 



1. 



This action is in response to the application filed on 4/1 1/2006. 



2. 



Claims 1-20 have been submitted for examination. 



Claim Rejections - 35 USC § 101 



3. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of 
matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the 
conditions and requirements of this title. 

4. Claims 1-7 and 11-12 are rejected under 35 U.S.C. 101 because the claimed 
invention is directed to non-statutory subject matter. The claims are directed to "a user 
interface description generating apparatus" which may be interpreted to be purely 
software per se where the subsequent limitations are simply components of that 
software. Software per se fails to meet the 35 U.S.C. 101 requirement that the 
invention be a "process, machine, manufacture, or composition of matter". The 
examiner suggests amending the subject claims to be placed on a tangible computer 
readable medium in order to alleviate this issue. 



5. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or deschbed as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill In the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner In which the Invention was made. 



Claim Rejections - 35 USC § 103 
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6. Claims 1, 2, 4-6, 8-10, 13, 14, 16, and 18-20 are rejected under 35 U.S.C. 103(a) 
as being unpatentable over Tomm (US 6,246,403) in view of Tran (US 6,054,990). 

7. As per claims 1,13, and 20, Tomm teaches the following: 

the sketch being representative of a graphic user interface (GUI) to be created, 
(column 1, lines 41-43), i.e. drawing programs are currently available on the market 
which enable a person to create a GUI bv drawing the desired GUI; 

a hierarchy determiner for determining, from the conformed versions, a hierarchy 
of relative containment among said conformed versions, (column 7, lines 9-1 1 ), i.e. 
once the list of boxes has been generated, the collection of boxes is organized into a 
data tree structure , as indicated by block 70; and 

a description generator for generating, from said hierarchy, a user interface 
description for creating the GUI, (abstract), once the data tree structure has been 
created, a GUI generating software program can use the data tree structure to generate 
the GUI . 

However, Tomm does not explicitly teach the inner workings of using a sketch 
recognition method for input. Tran teaches the following: 

a sketch identifier for examining a manual sketch of objects to identify 
sketched versions of the objects, (abstract), i.e. an object classification and recognition 
capability is provided to allow the present invention to recognize obiects being sketched 
by the user; 
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a sketch normalizer for conforming the identified sketched versions to resemble 
respective reference images, (abstract), i.e. an object classification and recognition 
capability is provided to allow the present invention to recognize objects being sketched 
by the user and convert the hand sketches into computer aided design like drawings; 

It would have been obvious to one of ordinary skill in the art to have 
modified Tomm's GUI design method with the sketch recognition method of Iran. One 
of ordinary skill in the art would have been motivated to have made such modifications 
because Tomm directly teaches in their background of invention in column 1 , lines 41- 
57, that at the time it was well known in the art that their existed GUI creating programs 
which accepted user input through a drawing program. Furthermore, as Tomm teaches 
in their abstract, their text file which they use to generate a GUI comprises a "two- 
dimensional pictorial representation". As was well known in the art, CAD drawings, of 
which Iran is directed towards, may be represented as a two-dimensional pictorial 
representation. Still further, as Tomm teaches in their abstract, their method generates 
a GUI from a text file. Tran teaches in column 7, line 66 - column 8, line 1 , that as 
"drawn" data is sent to their handwriting recognizer, the written instructions are 
converted into text data. 

8. Regarding claim 2, modified Tomm teaches the method of claim 1 as described 
above. Tran further teaches the following: 

said reference images comprise vector images, (column 14, lines 21-23), i.e. 
from step 153 to 155, the routine next determines whether the drawing is to be saved as 
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a hand-drawn bitmap sketch or as a converted series of vectors and/or predefined 
objects in step 156. 

9. Regarding claim 4, modified Tomm teaches the method of claim 1 as described 
above. Iran further teaches the following: 

the manual sketch includes characters, and wherein the sketch identifier is 
configured for applying optical character recognition (OCR), (column 8, lines 5-7), i.e. 
with suitable recognition software, text and numeric information can be entered into the 
pen-based computer system in a similar fashion. 

10. Regarding claim 5, modified Tomm teaches the method of claim 1 as described 
above. Tomm further teaches the following: 

said description generator is further configured for generating the user interface 
description to contain a layout of said conformed versions, (abstract), i.e. the GUI 
constructing program scans the text file and determines the x and v coordinates of the 
corners of all of the boxes and the data type associated with each box. 

1 1 . Regarding claim 6, modified Tomm teaches the method of claim 1 as described 
above. Tomm further teaches the following: 

the description generator is configured to generate the user interface description 
into a format specific to a target platform for the GUI, (column 3, lines 37-43), i.e. it 
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should also be noted that the GUI constructing program of the present invention is 
suitable for use with different types of windowing systems . 

12. Regarding claim 8, modified Tomm teaches the method of claim 1 as described 
above. Iran further teaches the following: 

an electronic storage medium, (column 6, lines 57-59), i.e. the processor 20 is 
also connected to a random access memory 22 for storing various run-time variables 
and data arrays; 

a liand-lield pen for creating tlie sketch, (column 7, lines 31-35), i.e. in one 
embodiment where the computer accepts handwritings as an input medium from the 
user, a combination pen/ digitizer unit 25, and a display LCD panel 26 having a viewing 
screen exposed along one of the planar sides of the enclosure are provided; and 

a digitizer for recording into tlie medium ttie sl^etch in real time as the sketch is 
being created, (column 7, lines 31-35), i.e. in one embodiment where the computer 
accepts handwritings as an input medium from the user, a combination pen /digitizer unit 
25, and a display LCD panel 26 having a viewing screen exposed along one of the 
planar sides of the enclosure are provided. 



1 3. Regarding claim 9, modified Tomm teaches the method of claim 8 as described 
above. Tomm further teaches the following: 



Application/Control Number: 10/575,575 Page 7 

Art Unit: 2176 

the apparatus stores in said medium a normalized sketch comprising the 
conformed versions, (column 3, lines 64-66), i.e. once the text file has been created, the 
text file is stored in a memory device which may be comprised by computer 4. 

However, as described above, Tomm does not explicitly describe the inner 
workings of a sketch recognition method for input. Tran further teaches the following: 

said apparatus further comprising a sketch editor for editing said normalized 
sketch stored in said medium, (column 11, lines 34-37), i.e. in Fig. 2, the menu 100 
allows the user to control the sketching system . Preferably, the menu 100 is a pull- 
down menu with a number of choices, including File, Edit , Object and View; 

said digitizer being configured for augmenting, according to input from the pen, 
said normalized sketch stored in said medium, (column 12, lines 32-36), i.e. referring 
now to Fig. 3, the operation of the magnifier 108 of Fig. 2 is disclosed. Upon beginning 
the process in step 120 of Fig. 2, the routine displays a magnifier icon whenever the pen 
touches the LCD screen 26 of the computer system. 

14. Regarding claim 10, modified Tomm teaches the method of claim 1 as described 
above. Tran further teaches the following: 

an electronic storage medium for storing said reference images, (column 7, lines 
2-3), i.e. the RAM 22 stores a database of the drawings of the present invention; and 

wherein the sketch identifier is configured for using the stored reference images 
in identifying said sketched versions, (column 13, lines 44-49), i.e. the routine of Fig. 5 
will compare the sketches against a library containing definitions and symbols for 
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predetermined objects if tlie user does not specify in advance that her or she is writing 
text, in such case the handwriting recognition engine is to be applied. 

1 5. Regarding claim 1 4, modified Tomm teaches the method of claim 1 3 as 
described above. Tran further teaches the following: 

the sketching step further includes the step of sketching, as an annotation to at 
least one of the objects, a label of a function of the object in said GUI, (column 4, lines 
49-57), i.e. each box also has a name and a data tvpe associated with it . which are text 
strings that are used by the windowing system executing the GUI construction program 
to create windows and other GUI components. The name is simply a word or a 
combination of words that will be displayed in the GUI window associated with the box. 
The data type informs the GUI constructing program of the manner in which the box 
must be handled by the GUI constructing program. 

16. Regarding claim 16, modified Tomm teaches the method of claim 13 as 
described above. Tran further teaches the following: 

at least one of the sketched versions of an object intersects another sketched 
version of an object, (column 6, lines 1-4), i.e. this data tree structure contains the 
relative dimensions and locations of the boxes that appear in the input text file, the 
names and types of the boxes, and the enclosure relationships among the boxes : and 

wherein the sketching step further includes the step of sketching, as an 
annotation to at least one of two nnutually intersecting ones of the versions, a label of a 
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function of the respective object in said GUI, (column 4, lines 49-57), i.e. each box also 
has a name and a data type associated with it . which are text strings that are used by 
the windowing system executing the GUI construction program to create windows and 
other GUI components. The name is simply a word or a combination of words that will 
be displayed In the GUI window associated with the box. The data type Informs the GUI 
constructing program of the manner in which the box must be handled bv the GUI 
constructing program. 

17. Regarding claim 18, modified Tomm teaches the method of claim 13 as 
described above. Tran further teaches the following: 

the sketching step further comprises the steps of: 

manipulating a pen by hand to create the sketch, (abstract), i.e. an object 
classification and recognition capability Is provided to allow the present invention to 
recognize oblects being sketched bv the user : and 

recording into the medium the sketch in real time as the sketch is being created, 
(column 7, lines 60-63), i.e. as the CPU 20 senses the position and movement of the 
stylus, It generates a corresponding Image on the screen to create the illusion that the 
pen or stvlus is drawing the image directiv upon the screen . 



1 8. Regarding claim 1 9, modified Tomm teaches the method of claim 1 3 as 
described above. Tran further teaches the following: 
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the step of pre-storing said reference images to aid in the identification performed 
in the examining step, (column 13, lines 44-49), i.e. the routine of Fig. 5 will compare 
the sketches against a library containing definitions and symbols for predetermined 
obiects if the user does not specify in advance that her or she is writing text, in such 
case the handwriting recognition engine is to be applied. 

19. Claim 3 is rejected under 35 U.S.C. 103(a) as being unpatentable over modified 
Tomm as applied to claim 1 above, and further in view of Eller et al. (US 5,287,417), 
hereinafter Eller. 

20. Regarding claim 3, modified Tomm teaches the method of claim 1 as described 
above. However, neither Tomm nor Tran explicitly teaches refining the sketches. Eller 
teaches the following: 

the sketch normalizer is configured for straightening lines and making lines 
mutually parallel, (column 3, lines 59-61), i.e. after the objects are recognized, they are 
displayed with perfected angles and lines . 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have further modified the sketch recognition method of Tomm in 
view of Tran, with the refining method of Eller. One of ordinary skill in the art would 
have been motivated to have made such further modifications because Tomm directly 
teaches in their background of invention in column 1 , lines 41-57, that at the time it was 
well known in the art that their existed GUI creating programs which accepted user input 
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through a drawing program. Furthermore, both Iran and Eller are analogous art in the 
field of interpreting hand drawn input. 

21 . Claim 7 is rejected under 35 U.S.C. 103(a) as being unpatentable over modified 
Tomm as applied to claim 1 above, and further in view of Carroll (US 2002/0085020). 

22. Regarding claim 7, modified Tomm teaches the method of claim 1 as described 
above. However, neither Tomm nor Tran explicitly teaches the description is written in a 
structured mark-up language. Carroll teaches the following: 

the description generator is configured for generating the description into a 
hierarchical, structured mark-up language, (abstract), i.e. an applications graphical user 
interface is specified using an XML document as an application interface file . At 
application compile time this application interface file is parsed, and the specifications 
therein used to retrieve graphical screen components from an interface library to create 
the user interface. 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have further modified the user interface document generation 
method of Tomm with the XML user interface document of Carroll. One of ordinary skill 
in the art would have been motivated to have made such modifications because both 
Tomm and Carroll are analogous art in the field of generating user interfaces from 
document files. 
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23. Claims 11, 12, 15 and 17 is rejected under 35 U.S.C. 103(a) as being 
unpatentable over modified Tomm as applied to claims 1,13 and 16 above, and further 
in view of Alexander (US 6,1 1 8,451 ). 

24. Regarding claims 1 1 and 15, modified Tomm teaches the method of claims 1 and 
1 3 as described above. 

the description generator is configured to generate tlie user interface description 
to reflect a stacking order based on an annotation to a sketched version of an object in 
said sketch, (column 4, line 67 - column 5, lines 5), i.e. the boxes 23 and 25 are dialog 
boxes containing GUI components commonly referred to as push buttons. The 
indication "button" on the top edges of the boxes 23 and 25 corresponds to the data 
type of the boxes and indicates to the GUI constructing program that the box data type 
is push button. 

However, Tomm does not explicitly teach a method of the annotation to the 
sketched version being an indication of a priority: 

said annotation indicating a priority for the annotated object with respect to at 
least one other of the objects as to which of two objects has priority to overlay the other 
of the two in said GUI, (column 24, lines 53-57), i.e. the Z order list data structure 612 
contains an identification 722 of all dialogs currently being managed by the dialog 
manager 402. Associated with each is a Z number 724 identifying the position in the 
visual sequence on the display screen. 
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It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have modified the indications of Tomm to further include the "Z 
number" value of Alexander. It would have been obvious to one of ordinary skill in the 
art to have made such modifications because Tomm directly teaches that the x and y 
coordinates of each box are stored In a list in memory in column 6, lines 41-45, and 
Alexander teaches of a similar list of storing the z value of each box in column 24, lines 
53-58. Alexander also teaches in column 24, lines 45-48, that "the Z axis is normal to 
the two-dimensional display screen. The Z order identifies the order of the displayed 
dialog boxes as they are layered on the display screen". Tomm shows of a desire for 
layering their boxes as may be seen in their teaching of column 6, lines 1-4, which 
teaches that their data tree contains "enclosure relationships", which is clear to one of 
ordinary skill in the art to be a form of overlay. This interpretation is further reinforced in 
Tomm's showing of Fig. 2B, where Tomm shows three different boxes (22, 23, and 25) 
"overlaying" a larger box (21). Still further, Tomm and Alexander are analogous art In 
the field of dialogue boxes. 

25. Regarding claim 12, modified Tomm teaches the method of claim 1 1 as 
described above. Tomm further teaches the following: 

said apparatus being furttier configured to recognize that said annotation 
indicates priority based on a dividing line within said sketched version of an object, 
(column 4, lines 43-45), i.e. it will be assumed herein that if a point lies on the top or 
right edges 17 and 19 of the box 10, the box 10 encloses the point, (column 4, line 67 - 
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column 5, lines 5), i.e. the boxes 23 and 25 are dialog boxes containing GUI 
components commonly referred to as push buttons. The indication "button" on the top 
edges of the boxes 23 and 25 corresponds to the data type of the boxes and indicates 
to the GUI constructing program that the box data type is push button. 

The examiner interprets Tomm's teaching of a top edge line which visually 
defines a box to encompass applicant's limitation of a "dividing line". 

26. Regarding claim 1 7, modified Tomm teaches the method of claim 1 6 as 
described above. Tomm teaches the following: 

the hierarchy determining step relatively positions in said hierarchy respective 
objects of said two mutually intersecting ones, (column 6, lines 1-4), i.e. this data tree 
structure contains the relative dimensions and locations of the boxes that appear in the 
input text file, the names and types of the boxes, and the enclosure relationships among 
boxes. 

As was well known in the art at the time, in a data tree structure, child nodes are 
connected to their parents, making them relatively positions in the hierarchy. 

However, Tomm does not explicitly teach a method of relatively positioning 
boxes in the hierarchy based on an annotation . Alexander teaches the following: 

based on an annotation created in the annotation sketching step, (column 24, 
lines 53-57), i.e. the Z order list data structure 612 contains an identification 722 of all 
dialogs currently being managed by the dialog manager 402. Associated with each is a 
Z number 724 identifying the position in the visual sequence on the display screen. 
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As described above in the rejections of claims 1 1 and 15 above, it would have 
been obvious to one of ordinary skill in the art to have modified the indication of Tomm 
to further include the Z value of Alexander. As Tomm's method already bases their 
hierarchical tree on "enclosure relations", e.g. overlays, it would have been obvious to 
one of ordinary skill in the art to have used the Z values of modified Tomm to position 
boxes in their hierarchical tree. 

Conclusion 

27. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure: 

-Bourgeois et al. (US 5,060,170), space allocation and positioning method for 
screen display regions in a variable windowing system. 

-Watanabe et al. (US 5,068,804), document input method and apparatus. 

-Geary et al. (US 5,206,950), software development system and method using 
expanding outline interface. 

-Hoffman et al. (US 5,347,627), graphical user interface including dynamic sizing 
and spacing. 

-Joseph (US 5,721 ,848), method and apparatus for building efficient and flexible 
geometry management widget classes. 

-Ulrich (US 5,917,487), data-driven method and system for drawing user 
interface objects. 

-Okada (US 5,956,029), user interface conversion method and apparatus. 
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-Cain et al. (US 6,014,138), development system with methods for improved 
visual programming with hierarchical object explorer. 

-Bolnick et al. (US 6,043,817), method and apparatus for arranging displayed 
graphical representations on a computer interface. 

-Scarborough et al. (US 6,353,448), graphic user interface display method. 

-Craycroft et al. (US 6,731,310), switching between appearance/behavior themes 
in graphical user interfaces. 

-Audleman et al. (US 6,806,890), generating a graphical user interface from a 
command syntax for managing multiple computer systems as one computer system. 

-Silverbrook et al. (US 7,134,601), method of generating a user interface for a 
computer system. 

-Silverbrook et al. (US 7,322,524), graphic design software using an interface 
surface. 

-Yen et al. (US 2002/0035595), method and system for generating user 
interfaces. 

-Blume (US 2004/0056900), system and method for window priority rendering. 
-Tobita et al. (US 2005/0062740), user interface method and apparatus, and 
computer program. 

-Horiguchi (JP 2005100308 A), efficient multiwindow GUI drawing algorithm to be 
driven at high speed even in small resource environment. 
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28. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to GREGORY A. DISTEFANO whose telephone number is 
(571)270-1644. The examiner can normally be reached on 7:30am-5:00pm Mon.- 
Thurs. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Doug Hutton can be reached on (571)272-4137. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 
Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published 
applications may be obtained from either Private PAIR or Public PAIR. Status 
information for unpublished applications is available through Private PAIR only. For 
more information about the PAIR system, see http://pair-direct.uspto.gov. Should you 
have questions on access to the Private PAIR system, contact the Electronic Business 
Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO 
Customer Service Representative or access to the automated information system, call 
800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

GAD 

3/15/2008 

/William L. Bashore/ 
William L. Bashore 
Primary Examiner 
Tech Center 2100 



